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@ Passive ranging and rapid autofocusing. 

(g) Apparatus and nnethods based on signal processing techniques are disclosed for determining the distance of 
an object from a camera, rapid autofocusing of a camera, and obtaining focused pictures from blurred pictures 
produced by a camera. The apparatus of the present invention includes a camera characterized by a set of four 
camera parameters: position of the image detector or film inside the camera, focal length of the optical system 
in the camera, the size of the aperture of the camera, and the characteristics of the light filter in the camera. In 
the method of the present invention, at least two images of the object are recorded with different values for the 
set of camera parameters. The two images are converted to a standard format to obtain two normalized images. 
The values of the camera parameters and the normalized images are substituted into an equation obtained by 
equating two expressions for the focused image of the object. The two expressions for the focused image are 
based on a new deconvolution formula which requires computing only the derivatives of the normalized images 
and a set of weight parameters dependent on the camera parameters and the point spread function of the 
camera. In particular, the deconvolution formula does not involve any Fourier transforms and therefore the 
present invention has significant advantages over prior art. The equation which results from equating two 
expressions for the focused image of the object is solved to obtain a set of solutions for the distance of the 
object. A third image of the object is then recorded with new values for the set of camera parameters. The 
solution for distance which is consistent with the third image and the new values for the camera parameters is 
determined to obtain the distance of the object. Based on the distance of the object, a set of values is 
determined for the camera parameters for focusing the object. The camera parameters are then set equal to 
these values to accomplish autofocusing. After determining the distance of the object, the focused image of the 
object is obtained using the deconvolution formula. A generalized version of the method of determining the 
distance of an object can be used to determine one or more unknown camera parameters. This generalized 
version is also applicabte to any linear shift-invariant system for system parameter estimation and signal 
restoration. 
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1 Field of Invention 

The present invention relates to an apparatus and a nriethod based on signal processing techniques for 
passive ranging of objects by a camera, i.e. determining the distance of objects from the camera without 
5 employing active methods such as sending out infrared, sonar, laser or other type of beams towards the 
objects, but using only ambient illumination. The present invention also relates to passive methods for rapid 
autofocusing of a camera, and obtaining improved focus images from blurred images produced by a 
camera. In general, this invention relates to system parameter estimatior) and signal restoration in any 
//near shlft-invar/ant system, i.e. a system that performs convolution operation on the input signal. 

2 Prior Art 

Rndinig the distance of objects from a camera system has applications in the fields of robotic vision, 
machine vision, autonomous vehicle navigation, and rapid autofocusing of camera systems. Many methods 
and apparatus are known for determining the distance of objects from a camera system. A review of the 
methods relevant to the present invention can be found in the following two patent applications filed by this 
same inventor: (i) U. S. patent application no. 126407. pending, filed on November 27,1987, and (ii) U.S. 
patent application no. 07/373996, pending, filed on 29th June, 1989. 

In all of known prior art (e. g. U. S. patent application no. 126407 (11/27/87) and 07/373996 (6/24/89) 
filed by this inventor, Japanese patent no. 62-284314 granted to FUJI in 1987, and Japanese patent no. 63- 
127217 (laid open date 5/31/88) awarded to CANNON) there are many common important drawbacks, in 
the relevant prior art tiie metfiods of determining distance are ali based on computing the Fourier 
transforms of two or more recorded images and then computing the rstio of these two Fourier 
transforms. Computing the two-dimensional Fourier transforms of recorded images is computationally very 
expensive. It involves complex and expensive hardware. 

Another drawback of the methods of prior art is that they cannot be used to determine the distance of 
small objects. They can be used to determine the distance of only large objects. This Is because the 
reliable computation of Fourier coefficients of the image of an object requires that the image of the object 
be large. 

Determining the distance of an object is an essential step In any method of autofocusing, and also in 
any method of obtaining focused images from blurred images of the object. Therefore, the drawbacks 
mentioned above are also applicable to prior art on autofocusing and methods for obtaining improved focus 
images from blurred images. 

35 3 Summary of the Invention 

An object of the present invention is a method of finding the distance of an object from a camera 
system by recording and processing at least two pictures of the object. More than two pictures may be 
used to avoid ambiguity and to increase robustness. 
40 Another object of the present invention is a method for rapid autofocusing of cameras by acquiring and 
processing three or more images. 

An additional object of the present invention is a method for obtaining improved focus images from 
blurred images. 

A further object of the present invention is a method for estimating system parameters and recovering 
45 input signal in any instrument that can be modeled as a linear shift-invariant system, i.e. a system that 
performs convolution operation. 

The apparatus of the present invention comprises a camera system including (i) an optical system for 
forming images, (ii) an image detector for recording images, (iii) a camera aperture through which light 
enters the camera, and (iv) a light filter to control the spectral content of light entering the camera system. 
50 The optical system is characterized by a first principal piane and a second principal plane. The second 
principal plane is closer to the image detector than the first principal plane. 

The camera system is characterized by a set of camera parameters which include: 

(i) the distance between the second principal plane and the image detector, 

(ii) the diameter of the camera aperture, 

55 (iii) the focal length of the optical system, and 

(iv) the spectral transmittance characteristic of the light filter. 

The apparatus also includes a camera controller for setting the values of the camera parameters to 
desired values, and an image normalization means. The image normalization means is used to convert 
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different images of an object, each image recorded with different settings of camera parameter values, to a 
standard format. The standard format is such that an image of an object represented in this format can be 
expressed as the result of convolving a standard focused Image of the object with the point spread function 
of the camera. An image represented in the standard format is also referred to as a normalized image. 
5 The apparatus further includes a derivative computation means for computing derivatives of normalized 

images, an equation solving means, and a set intersection means. The equation solving means can 
substitute into an equation the values of different derivatives of normalized images and camera parameter 
values, and solve the resulting equation to obtain a set of solutions for the distance of the object from the 
camera. The set intersection means is used to take two sets of solutions for the distance of the object from 
10 the camera and intersect the two sets to find the common solution to the distance of the object. 

In the apparatus for rapid autofocusing, a focus parameter determination means is included in addition 
to the apparatus for determining the distance of the object. The focus parameter determination means is 
used to determine values for the camera parameters based on the distance of the object to be focused so 
that the object is focused when the camera parameters are set to the values thus determined. 
76 The method of determining the distance of an object from a camera includes the steps below. 

The camera parameters of the camera are set to a first set of camera parameter values by the camera 
controller. Then a first image of the object is recorded. The camera parameters are then set to a second set 
of camera parameter values by the camera controller, and a second image of the object is recorded. 

The first image is preprocessed to obtain a first normalized image. The first normalized image is such 
20 that it is equal to the convolution of a focused image of the object with a first point spread function of the 
camera characterized by the first set of camera parameter values. The second image is preprocessed in a 
manner similar to the first image to obtain a second normalized image. 

A set of derivatives of the first normalized image is computed to obtain a first set of image derivatives. 
Similarly, a set of derivatives of the second normalized image is computed to obtain a second set of image 
25 derivatives. 

The values of the first set of camera parameter values.- the second set of camera parameter values, the 
first set of image derivatives, and the second set of image derivatives are all substituted into a predeter- 
mined equation. This equation is predetermined by equating a first expression to a second expression 
where the first expression gives the focused image of the object in terms of the first set of the camera 
30 parameter values and the first set of image derivatives, and the second expression also gives the focused 
image of the object but in terms of the second set of camera parameter values and the second set of image 
derivatives. After the substitution into the predetermined equation, the resulting equation contains only one 
unknown which is the distance of the object from the camera. This equation is then solved to obtain a first 
set of solutions for the distance. This set typically contains two members. In obtaining the first set of 
35 solutions for distance, no Fourier transforms are computed. In comparison, prior art involves the 
computation of Fourier transforms which is a complex and expensive operation. Thus the method of present 
invention has a great advantage over prior art. 

Next the camera parameters are set to a third set of camera parameter values by the camera controller 
and a third image is formed inside the camera. This third image is recorded. Then the member of the first 
40 set of solutions which is consistent with the third image and the third set of camera parameter values is 
determined to obtain the distance of the object from the camera. The consistency check can be performed 
in many ways. For example, the second and the third images can be used to obtain a second set of 
solutions for the distance, and then the first set of solutions and the second set of solutions can be 
intersected to obtain a common solution which gives the distance of the object. 
45 In the method of rapid autofocusing, first the distance of the object to be focused is determined 
according to the above method. Then, based on the distance of the object and and an image focusing 
formula, a new set of camera parameter values is determined. The camera parameters are set by the 
camera controller to the new set of camera parameter values, thereby autofocusing the object. 

The method of determining an improved-focus image of an object from its blurred images recorded with 
50 different camera settings can.be summarized as follows. First the distance of the object is determined by 
using the method outlined earlier. Based on this distance, the moment parameters of the point spread 
function corresponding to one of the recorded image of the object is determined. This image and its 
derivatives are then substituted into a deconvolution formula to obtain an improved focus Image of the 
object. 

55 

4 Advantages 

An important advantage of the present invention is that it does not involve the computation of 
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Fourier transforms whereas prior art does invoive the computation of Fourier traris forms. Computation 
of Fourier transforms is computationally very expensive. This operation is slow and requires expensive 
hardware. Therefore, the apparatus of the present invention is cheaper, simpler, and faster than the 
apparatus of prior art. 

5 Another important advantage of the present invention is that it can be used to determine the distance of 

very small objects. This capability is lacking in prior art. In the present invention, the determination of 
distance of small objects is possible because the computations are based on the derivatives of images. The 
derivatives of the image of an object can be computed even when the image of the object is relatively 
small. The lack of this capability in prior art is due to the fact that the methods of prior art are based on the 

10 computation of Fourier transforms. The reliable computation of the Fourier transform of the image of an 
object requires the object to be relatively large. 

The present invention shares many important features of an earlier invention by this same inventor ( U. 
S. patent application no. 07/373996, filed on June 29th, 1989, pending). For example, the present invention 
is not restricted by the form of the point spread function of the camera. It is applicable to any camera 

75 system that can be modeled as a linear shift-invariant system (i.e. a system that performs convolution). 
Therefore it Is applicable even in the presence of significant optica! aberrations and image distortion caused 
by diffraction. In particular, the present invention is applicable but not restricted to Gaussian and cylindrical 
shaped point spread functions. In the present invention, the point spread function is characterized 
completely in terms of the relevant camera parameters, thus accounting for diffraction effects and lens 

20 aberrations. 

The present invention Is also general and therefore many camera parameters can be changed 
simultaneously instead of changing only one camera parameter at a time. Further, it generalizes the method 
of determining distance to determining other camera parameters. It also generalizes the method to any 
linear shift-invariant system for estimating any number of system parameters and estimating the input 
25 signal. 

Further advantages of the present invention wil! become apparent hereinafter to those with ordinary skill 
in the art to which the present invention pertains. 

5 Brief Description of the Drawings 

30 

For a further understanding of the objects of the present invention, reference is made to the following 
description of the preferred embodiment which is to be taken in connection with the accompanying 
drawings. 

Rg. 1 is a block diagram of the camera system of the preferred embodiment. It illustrates a focal length 
35 controller, image "detector position controller, a camera aperture diameter controller, and a filter to control 
the spectral content of light entering the camera; 

Rg. 2 illustrates the camera parameters and the image formation process in the preferred embodiment 
of the camera system . 

Rg. 3 is a block diagram of an alternative camera embodiment having two cameras (instead of one 
40 camera) with different camera parameter values and a beam-splitting arrangement for viewing a scene 
through a common aperture. 

Fig. 4a is a schematic drawing showing the apparatus of the present invention for finding the distance of 
an object from a camera. Fig. 4b is a continuation of this figure. 

Rg. 4b is a cpntinuation of Fig. 4a- This diagram illustrates the method of obtaining a unique solution for 
45 the distance of an object by intersecting two sets of solutions, and also the methods for rapid autofocusing 
and obtaining the focused image of an object from its blurred image. 

Fig. 5 shows an example where a digital image processing computer has been used in the implementa- 
tion of the apparatus of the present invention. 

Fig. 6 shows the camera's field-ot-view divided into many smaller subfield-of-views and the correspond- 
so ing subimages into which the image has been divided. 

Fig. 7 shows an implementation of a new deconvolution formula to obtain the focused image of an 
object from its blurred image. 

Rg. 8 shows the point spread function (or the impulse response function) of a misfocused camera with 
a circular aperture. 

55 

6 Detailed Description of the Preferred Embodiments 

Fig. 1 shows the camera system of the preferred embodiment. It includes a light filter LF, an aperture 
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stop AS, an optical system OS, and an innage detector ID. The light filter LF controls the spectral content of 
light entering the camera. For example, a red filter transmits only red light and blocks light of all other 
wavelengths. The amount of light entering the camera is controlled by the aperture stop AS. In the preferred 
embodiment, the aperture of the aperture stop is circular. Therefore, the amount of light is controlled by 
6 varying the diameter of the aperture of the aperture stop. The aperture of the aperture stop is also referred 
to as the camera aperture. In the preferred embodiment, the optical system OS consists of lenses. The 
optical system OS forms an image of the scene on the image detector ID. The image formed on the image 
detector ID is converted to an electronic video signal which forms the output of the camera system. 

The camera is characterized by a set of camera parameters. The camera parameters are {see Fig. 2): 
10 (i) the distance s between the second principal plane P2 of the optical system and the image detector ID, 
(ii) the diameter D of the camera aperture, (iii) the focal length f Of the optical system, and (iv) the spectral 
transmtttance characteristic X of the light filter LF. The values of the camera parameters s, f, and D, are 
respectively controlled by three controllers: image detector position controller, focal length controller,' and 
aperture controller. These controllers can be used to set the values of the camera parameters s, f, and D to 
75 desired values. The three controllers together constitute the camera controller. 

An alternative embodiment of the camera system is obtained by a minor modification of the preferred 
embodiment described above. In this modified embodiment, instead of using a single camera system which 
provides means for changing the camera parameters, multiple cameras are used. These multiple cameras 
are such that the camera parameter values of each individual camera differs in at least one of the camera 
20 parameters in comparison with all other cameras. That is. no two cameras among the multiple cameras 
have identical camera parameter values for all camera parameters. In this apparatus, there is no necessity 
to provide for varying any of the camera parameters of the individual cameras. All camera parameters of 
each camera can be fixed, thus avoiding the necessity for any moving parts in the camera system. 
Therefore the apparatus can be made rugged. Each camera is used in acquiring one image corresponding 
25 to one set of values of camera parameters. Each of the individual cameras view the scene through a 
common aperture with the aid of a beam-splitting arrangement. Except for the above differences, the 
alternative embodiment is similar to the preferred embodiment. Figi 3 shows an example where two 
cameras with different camera parameters values view the scene through a beam-splitter consisting of a 
half-silvered mirror and a full mirror. 
30 Fig. 4a and Fig. 4b show the apparatus of the preferred embodiment of the present invention. In these 
figures, the input from the scene to the camera are the focused image f(x,y) of the scene, and the 
geometry r(x,y) of the scene. The focused image f(x,y) specifies the amount of light received from a 
particular direction of the scene specified by (x,y). The geometry r(x,y) specifies the distance of the object 
from the camera along the direction specified by (x,yl The values of the set of camera parameters 
35 {X,s,r,D} are set by the camera controller. The output of the camera Is a recorded image 9(x,y). In Fig. 4a, 
the output of the camera corresponding to three different values = {Xi,Si,fi,Di}. ea2 = {X2.S2,/2,Z?2},' 
and = of the set of camera parameters {X,s,f,D} are denoted by g^(x,y), g2(x,y), and ga- 

(x,y) respectively. 

The apparatus also includes an image normalization means. The image normalization means is used 
40 to convert different images recorded with different values for the set of camera parameters {X,s,A,D} to a 
standard format. This conversion process is called image normalization. 

The image normalization is such that the image of an object after normalization can be expressed as 
the result of convolving a standard focused image of the object with the point spread function of the camera 
characterized by the corresponding camera parameter values. For example, in Fig. 4a. the images g^ (x,y), 
^5 g2(x,y) and g^(x,y) are recorded with different values Sai. e^a and Oas for the set of camera parameters 

The point spread function of the camera is denoted by h(x,y; e^Jo) where (x,y) denotes a position on 
the image of an object, = {Ksj,D} denotes the values of the set of camera parameters, and /b denotes 
the distance of the object along the direction specified by (x,y). A standard or normalized focused image fn- 
60 (Ky) corresponding to a focused image f(x,y) can be defined as 

fn(x,y) = ^ f(x,y) where (1) 



55 



E 




(2) 



Intuitively. fn(x.y) specifies the fraction of the total light incident on the camera aperture from the 
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direction specified by (xy). 

The three images g^(x,y), gzCx.y) and g3(x,y) after normalization are denoted by 9n^(x,yl gn2(^fy) and 
gn3(x,y) respectively. Therefore, according to the definition of the image normalization operation. 

5 gnA^y) h(x,y;e^^,ro)* fn(x,y) (3) 

9n2(x,y) = h(x,y;ea2,ro) * fn(x,y) (4) 

gn3(Ky) = h(x,y;ee2,ro) * fn(x,y) (5) 

10 

where * denotes the convolution operation. The convolution operation can be defined in expanded form 
using equation (3) as an example as below: 

^ni(x,y)= / / h{x - - 7]]eaiyTo) f{(,v)ct^ (irf . (6) 

J — oo J — oo 

The apparatus also includes a derivative computation means for computing various derivatives of 
20 normalized images. For example, some of the derivatives of gn^(x,y) are 



5^ 5^ 
5^ni(a:,y), g^9rxi{^,y)j and ^^ni(x,y) . (7) 



25 

The apparatus further includes an equation solving means which can solve an equation of the form 
EE ^i.i,k-i{h,^ai,ro) gil^-' = E E t^2.i.*-.-(A,e„2,ro) gti^' (8) 

where iVi,,;i(./(/),eai,ro) and v^2j,k-i(^f^az^^o) represent expressions for weight parameters and 

35 

i.Ar-i 
9rxl 

and 

40 

are partial derivatives defined by 

45 

" ^ £i::7 5ni(a:,y), and (9) 



50 



dy 



In equation (8) all quantities except the distance of the object To corresponding to the image point (x,y) 
are known. Therefore, solving this equation, a set of solutions is obtained for To Typically the set of solutions 
55 for To contains two elements. 

The apparatus includes a set intersection means (see . Fig. 4b) to intersect two sets and find the 
elements common to both the sets. For example, if rs^ = {r^^,r^2} and rs2 = {/21 r22} are two sets, and r^2 
= r22 .then the set intersection means can intersect r^i and to give {/'12}. i.e. 
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r.if]r,2 = {ri2} = {r^j} . (U) 

The apparatus for rapid autofocusing (Fig. 4a and Fig. 4b) is similar to the apparatus for determining the 
distance of an object. The main difference is that the apparatus for rapid autofocusing has an additional 
component which is the focus parameter determination means shown In Fig. 4b. The focus parameter 
determination means takes as Input the distance of the object to be focused and gives as output a set of 
values i\nSfJf,Df} for the set of camera parameters {Ks,f,D} for focusing the object. The camera 
parameters of the camera are set to {XuSfJuDf} so that the object Is focused Inside the camera. 

The apparatus for obtaining the focused image of an object from its blurred Image (Rg. 4a and Fig. 4b) 
is also similar to the apparatus for determining the distance of an object. The only difference is that the 
apparatus for obtaining the focused image has an additional component called the focused image 
determination means. This component takes as input the various derivatives of the blurred image of the 
object, the camera parameter values, and the distance of the object, and substitutes these input quantities 
into a deconvolution formula to give as output the focused image of the object. 

Each component of the apparatus of the present invention can be Implemented as a special purpose 
hardware using analog and digital electronic circuitry. In particular, a digital computer such as a micropro- 
cessor can be used to implement the image normalization means, the derivative computation means, the 
equation solving means, and the set Intersection means. The focus parameter determination means and the 
focused image determination means can also be implemented using a digital computer. Fig. 5 shows an 
example where a digital image processing computer has been used to implement the apparatus of the 
present invention. 

Having described the apparatus of the present Invention, the method of the present invention will be 
described next. Fig. 4a and Fig. 4b are also helpful In understanding the method of the present invention. 

In order to determine the distance of an object from a camera, the set of camera parameters = 
{KsJ,D} is set to a first set of camera parameter values Cai = {Xi.Si./i .Di} by the camera controller, i. e. . 
the camera controller sets the camera setting so that X = Xi. s = Si, / = A. and D = . Then a first 
image g^{x,y) of the object is fornned inside the camera and the image is recorded. The set of camera 
parameters is then changed and set to a second set of camera parameter values = {K2.S2j2,!h} by 
the camera controller, i. e. , the camera controller changes the camera setting so that X = X2,s = S2J = 
fc.and D - D2- The second set of camera parameter values e^z Is such that it is different from the first set 
of camera parameter values in at least one element i. e. eai ^ 6^2 . i- e. (Xi X2) or (Si ^ S2) or (/1 fc) or 
(D^ ^ D2). Then a second image g2(x,y) of the object is formed Inside the camera and the image is 
recorded. 

The first image is preprocessed to obtain a first normalized Image g„i(x,>/). The first normalized 

image is such that it is equal to the convolution of a standard focused Image fn(x,y) of the object with a first 
point spread function h^{x,y,ee^,ro) characterized by the first set of camera parameter values ( and the 
distance ro of the object ). That is 

Pm(x,y) = h{x,ye^uro) * fn(x,}/}. (12) 



Conversion of the first image gi(x,>/) to the first normalized image QnAx,)/) is accomplished using the 
image normalization means. This conversion process is called image normalization. The image normaliza- 
tion process can include many operations such as correcting for the response of sensor elements in the 
image detector of the camera, correcting for non-uniform light transmission of the camera, correcting for 
changes In magnification, and correcting for changes in brightness. These operations are known in phor art 
and hence will not be elaborated here. For example, a detailed description of the image normalization 
operations can be found in a copending patent application filed by this same Inventor (U. S. patent 
application no. 07/373996. June 29,1969). 

The second Image g2(x,y) is also preprocessed In a manner similar to that of the first image g^{x,y) to 
obtain a second normalized image g„2(x,y). Therefore, the second normalized Image gnzix.y) Is such that it 
is equal to the convolution of the standard focused image /n(x,y) with a second point spread function fh 
(x,y;ea2./b) characterized by the second set of camera parameter values 632 (and the distance ro of the 
object): That is 

9n2(x,y} = h(x,yes2.ro) * /n(x.y). (13) 
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Conversion of the second image g2(>f,y) to the second normalized image gn2(>'f*y) is accomplished using 
the image normalization means. 
5 A set of derivatives of the first normalized image gni{x,y) is computed to obtain a first set of image 

derivatives 

7i = {^nf'Vi = 0, 1,2,...,A:, and A = 0,1,2,...,;V} (14) 

70 

where 



75 

is as defined in equation (9). For example, if A/ = 2, then 

f^O.O 0,1 ^1,0 ^2,0 1,1 0,2-1 /ic\ 
Vl = i9nl 1 9ni > 5nl J 9n.i i 9rxl ^ 9ni I • (15) 

20 

Similarly, a set of derivatives of the second normalized image Qnzi^M 's computed to obtain a second 
set of image derivatives 

= K'2"Vi = 0,1,2,... and fc = 0,1,2,.. .,yV} (16) 



where 

30 

i.Jt-t 

9rx2 



is as defined in equation (10). For example, if A/ = 2, then 

35 

f 0.0 0,1 _l,0 2,0 1.1 0.2i 
V2 = 1^„2 ^ 9rv2 J 9rv2 ) 5n2 ) ^n2 > 5n2 / • 1 1 0 

40 The computation of the first and second set of image derivatives ^i^ and 3/2 is accomplished using the 
derivative computation means. 

The values of the first set of camera parameter values eai, the second set of camera parameter values 
©32. the first set of image derivatives 771, and the second set of image derivatives 7^2. are all substituted into 
equation (8) (see Fig. 4a). 
45 The nature of equation (8) will be described now. 
Define 

fi = {w^J,f,.i/^ = 0.1,2. ...k, for k = 0.1,2, ...N (18) 
50 as a first set of weight parameters corresponding to the first set of image derivatives t/i , and define 
€2 = {iV2./.;c-, //■ = 0.1,2. ...k, for k = 0,1.2, ...N} (19) 

as the second set of weight parameters corresponding to the second set of image derivatives 7^2- For 
55 example, if A/ = 2, then 

fl = fl^1,0,0, 1^1,1,0, Wi, 0,1, IVi,2,0, 1^1,0,2,7 (20) 
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and 

^2 = (W2,0.0, W2,l,0. W2,0,1. IV2,2.0, iV2.l,1. tV2,0,2, (21) 



For each weight parameter W/.i/r-,- there is a corresponding image derivative 

For example, the weight parameters iVi,o,o, m/i,o,i. Wi,o,2i and iVi,2,o, respectively correspond to the image 
derivatives 

0.0 0,1 0.2 2,0 
9nl > 9nl i 9nl ? 9nl • 

Similarly, the weight parameters M/2,0.0, »V2,o,i, and w respectively correspond to the image 

derivatives 



0,0 0.1 0,2 2,0 

9n2 » 9n2 J 9ti2 J 9n2 ' 

Multiplying each member of the first set of image derivatives by its corresponding weight parameter, a 
first set of weighted image derivatives 

W^ = {w^j,k.i/i = 0.1.2, .../c. for k = 0J.2,...AA> (22) 

can be obtained. 

Similarly, multiplying each member of the second set of image derivatives V2 by its corresponding 
weight parameter, a second set of weighted derivatives 

^2 = {W2,i,H~i // = 0,1.2, .../C. for k = 0,1,2, ...AA} (23) 

can be obtained. 

Summing, all elements of the first set of weighted derivatives W-i gives a first expression which is equal 
to the left hand side of equation (8). Similarly, summing all elements of the second set of weighted 
derivatives W2 gives a second expression which is equal to the right hand side of equation (8). Therefore, 
equation (8) can indeed be obtained by equating the first expression to the second expression. The weight 
parameters Wi^/^^.,- are defined such that the first expression is equal to the focused image of the object 
expressed in terms of the first set of camera parameter values eai and the first set of image derivatives t^i . 
That is. the weight parameters Wij^t<-i are defined such that 



(24) 



Similarly, the weight parameters IV2././C-1 are defined such that the second expression is equal to the focused 
image of the object expressed in terms of the second set of camera parameter values 0^2 and the second 
set of image derivatives 1)2. That. is, the weight parameters W2j,k^ are defined such that 

AT k 

M^.y) = Z E ^^2"" • (25) 

k=0 t=0 

Recalling from equation (12) that gn^{x,y) is equal to the convolution of the first point spread function h- 
(x,y,Ba^.ro) with the focused image f/7(x,y), it should be recognized that equation (24) is actually a 

10 



k=0 i~0 
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deconvolution formula. The blurred image corresponding to gn^{x,y) is deconvolved to recover the focused 
image fn{x,y). A similar observation holds good for equations (13) and (25). This deconvolution formula is 
not known in prior art. This formula was first derived and is being disclosed in another U. S. patent 
application filed by this same inventor on July 3, 1990, having the title " Method and apparatus for restoring 
5 convolution degraded images and signals". In this patent, a procedure is also described for defining the 
weight parameters i^i,/.#c-i and Wzj^k-i- 

Each of the weight parameter w^j^h-i can be expressed in terms of a first set of moment parameters M^ 
defined as 

10 M-i = {h^j^},.i/i = 0.1,2. .../c,for/c = 0.1.2, ...A/} (26) 
where 



J — oo J — oo 

The first set of moment parameters Mi characterizes a first point spread function /7(x,y;eai,/b). The first 
point spread function is the point spread function of the camera when the camera parameters are set equal 
20 to the first set of camera parameter values, i.e. ea =- eai- Similarly each of the weight parameters Wz,-,,!,.-, can 
be expressed in terms of a second set of moment parameters M2 defined as 



(27) 



M2 = {h2,i.M/i = 0,1,2. ...k.ior k = 0,1,2, ...A/} (28) 
25 where 



h2,i,k-i = I / a:' y*- ' Ai(x,y;ea2,ro) (fx <iy . (29) 

» — OO J — oo 

30 

The second set of moment parameters M2 characterizes a second point spread function /?(x,y;ea2/o). 
The second point spread function is the point spread function of the camera when the camera parameters 
are set equal to the second set of camera parameter vaiues, i.e. eg = ea2- 

35 A method of deriving algebraic expressions for the weight parameters w^J^|^.i in terms of the first set of 
moment parameters M^ is described in earlier mentioned U.S. patent application filed by this inventor on 
July 3, 1990- The same method can be used to derive algebraic expressions for the weight parameters 
^zjM-'i terms of the second set of moment parameters A^2. Therefore, the algebraic expression for a 
weight parameter w^jj^.-, can be denoted in short form as w^i ,,>.,( Mi ). Similarly, the algebraic expression for a 

40 weight parameter iva./.k-/ can be denoted in short form as W2,i,k-A^2). For example, taking A/ = 2, the 
following algebraic expressions can be derived for w^j^^^,-, and W2j,k-i - 



45 



50 



55 
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25 



30 



45 



^1.0.0 ' 
1 

^2,0,0 = ^ 



^1.0,1 



*-2,0,0 



^2.0.1 

1^2.0.1 = Ti 1 

'^2,0,0 



0.0 

76 "-2,1,0 

"^2,0,0 



^1,0.2 



hi 

h 

^1,0,1 '^i 



0.2 



"-2,0,0 '^'*2.0,0 



■^1.14 



"-2,0,0 '^'*2.0,0 

A3 ~ A2 ' 

"-1.0,0 '''i.o,© 



_ ^2.0,1^2.1.0 '^2.1,1 

"^2.1,1 - — ri . 

'^2.0.0 "'2.0.0 



^?.i.o 


^1.2.0 


A3 

"1,0.0 




^2.1.0 


^2,2.0 


A3 

'^2.0.0 





ti;2.2.o = i3— -;rr2 — • (30) 



35 However, according to equations (26), (27) the first set of moment parameters Mi , is itself determined by 
the form of the; point spread function /}. the first set of camera parameter values Oai, and the distance To of 
the object from the camera. Therefore, the algebraic expression w^jj^^M^) can be denoted by w^j^^-r 
(/).eai,ro) to more explicitly convey the fact that the weight parameters Wi,/^./ can be expressed in terms of 
the point spread function h, the first set of camera parameter values Oai, and the distance /b of the object 

40 from the camera. For a similar reason, the algebraic expression iV2,/,/(./(W2) can be denoted by »V2,/,;(^- 
(/>,ea2.ro). 

In summary, equation (8) is formulated to obtain an equation which contains only one unknown which is 
the distance ro of the object from the camera. This equation Is solved to obtain a first set of solutions 
Typically Tji contains two elements and therefore can be represented as 



= {/•ii,ri2}. (31) 



Note that, in contrast with prior art, the first set of solutions ^s^ has been obtained without computing 
50 any Fourier transform. This constitutes an Important advantage of the present Invention. 

To obtain a unique solution for the distance of the object, additional information is needed. In the 
method of the present invention, additional information is obtained by setting the camera to a third set of 
camera parameter values 0^3 = {X3, S3. ^3,03} such that e^s ^ Sai and ^^z ( i.e. at least one of the 

camera parameter is different), forming an image of the object inside the camera, and recording a third 
55 image g3(x,>). Among the members of the first set of solutions only one member will be consistent with 
the third set of camera parameter values Bas and the third image g3(x,>). This member which is consistent 
with Bas and g3(x,y) Is determined to obtain the distance of the object from the camera. 

There are many methods of checking the consistency of a solution with Bas and g3(x,y). One method is 
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to use the third image g3(x,» and the second innage g2(x,y) and determine a second set of solutions r^z for 
the distance of the object from the camera. The method of determining r^z using g3(x,y) and g2(x,y) is very 
similar to the method that has been described above for determining the first set of solutions using the 
first image gi (x,>) and the second image g2(x,y). The second set of solutions fsz will typically contain two 
5 elements. One of the elements of r^z will also be an element of rji. This element is determined by taking 
the intersection of the sets r^^ and r^z- This common element gives a unique solution for the distance of the 
object from the cannera. Thus the distance ro of an object from a camera can be determined. 

In principle, acquiring and processing just three images of an object are sufficient to determine the 
distance of the object. However, a more robust estimate of the distance can be obtained by processing 

10 several distinct triplets of images of the object. Each distinct triplet of images is processed to obtain one 
independent estimate of distance. A weighted average of all the independent estimates of distances yields a 
robust estimate of distance. In general, the distance of the object estimated by the above described method 
can be combined with other independent distance estimates (obtained by either the same hnethod or a 
different method) by taking a weighted average to obtain a more robust estimate of the distance. 

75 The above method of determining the distance of an object from a camera can be applied to all objects 
in the field-of-view of the camera simultaneously to obtain the distance of all objects in the field-of-view. For 
example, the field-of-view of the camera can be divided into many smaller subfield-of-views and in each 
subfield-of-view the distance of the object in that subfteld-of-view can be determined. Dividing the camera's 
field-of-view into many subfield-of-views corresponds to dividing the image formed on the image detector 

20 into many smaller subimages where each subimage corresponds to one subfield-of-view. Fig. 6 illustrates 
this concept- Each subimage can be processed in parallel with other subimages to determine the distance 
of the object in the corresponding subfield-of-view. Thus, the distance of all objects in the field-of-view 
of a camera can be determined simultaneously from only three images. This method has important 
applications in robotic vision where it is required to find the position of all objects in a scene very fast with 

25 minimal computation. An array of numbers representing the distance (or depth) of object points in a scene 
in small subfields-of-view of a camera is called the depth-map of the scene. Therefore, the method of the 
present invention is useful in obtaining the depth-map of a scene in an efficient nnanner. 

The present invention also includes a method for rapid autofocusing of a camera (see Fig. 4a and Rg. 
4b). In this method, first the distance of the object to be focused is determined by using the above method 

30 for determining the distance To of an object from the camera. Based on distance To and an image focusing 
formula, a set of camera parameter values Bar = {\f,Sf,ff,Df} is determined which corresponds to focusing 
the desired object. An example of the image focusing formula is the well-known lens formula ^/f = 1/t/ + 
1/1/ (see Fig. 2) which relates the focal length of the camera to the distance of the object and the distance of 
its focused image. The relation between this image focusing formula and the set of camera parameters ear 

35 is given by equation (58) which is explained later. The camera is then adjusted so that the camera 
parameters are equal to the values determined by Qef- This accomplishes autofocusing. 

In the method of obtaining the focused image of an object from its blurred image (see Fig. 4a and Fig. 
4b), first the distance /b of the object is determined by employing the method described above. Then this 
distance information is used to compute the weight parameters iv?,/^-,-. The image derivatives 

40 

of the first normalized image are computed next. The values of the weight parameters w^jf,.i and the image 
45 derivatives 

9rxl 

50 are then substituted into the deconvolution formula given by equation (24) to obtain the focused image of 
the object. Alternatively; equation (25) can also be used. Fig. 7 illustrates an implementation of equation (24) 
for deconvolution with N = 2, g(x,y) = gni(^.y). = w^jj and f{x,y} = fn(x,y). 

7 Paraxial Geometric Optics 

55 

There are many models for the point spread function of a camera. The method of the present invention 
is applicable to all models. To illustrate the method of the present invention, a model of the point spread 
function derived from paraxial geometric optics is considered. 
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It is known in prior art that according to paraxial geometric optics, the point spread function of a camera 
has a non-zero constant value equal to MA inside a region whose area is equal to A and outside this region 
the value of the point spread function Is equal to zero. The shape of the region is the same as the shape of 
the aperture of the camera, except for a magnification factor. A discussion of the point spread function of a 

5 camera based on paraxial geometric optics principles can be found in the technical report: " Determining 
distance from defocused images of simple objects TR 89.07.20, Computer Vision Laboratory, Department 
of Electrical Engineering, State University of New York, Stony Brook. NY 11794-2350, 20th July 1989. 
authored by this same inventor. 

In the preferred embodiment of the present invention, the shape of the camera aperture is circular. 

70 Therefore the point spread function has a non-zero constant value inside a circular region and has a zero 
value outside of the circular region. Fig. 8 illustrates the point spread function in this case. This circular 
region is also referred to as "blur circle". The point spread function in this case is circularly symmetric 
about the center of the blur circle. If the camera is set to the first set of camera parameter values, Cai = 
{Xi,Si,/i,Dt} and /b is the distance of the object, then it is known in prior art that the radius of the blur circle 

75 Ai can be expressed in reduced units as 



20 




If Ai is the radius of the blur circle expressed in absolute units, then the relation between Ai and Ai' is 



25 



(33) 



For example, see technical report: "On the depth information in the point spread function of a defocused 
30 optical system", TR 90.02.07, Computer Vision Laboratory. Department of Electrical Engineering. State 
University of New York, Stony Brook. NY 11794-2350, February 7. 1990. authored by this same inventor. 
For this case, the first point spread function /)(x,y;eai./b) is given by (see Rg. 8) 



35 



I 



if -h < 
0 otherwise. 



(34) 



40 



In the above equation, we can substitute for Ai from equation (32) to obtain a more explicit expression 
for /?(x,y;eai.ro) directly in terms of the set of camera parameters Oai and the distance /b of the object. 
The (/T7,/7)th moment of the first point spread function /7(x.y;eai,ro) is denoted by and defined as 



45 



-OO J—co 



(35) 



50 



It can be shown that all odd moments of /)(x,y;eai»ro) are zero, i. e. 
^i.m.n = 0 for {m odd) or (n odd). (36) 



In fact it can be easily shown that all odd moments of any circularly symmetric function is zero (a function 
h{x,y) Is circularly symmetric if /7(x,y) is a constant when x^ + remains a constant). The even moments 
66 of the point spread function h{x,y;Bs^»ro) can be shown to be 



14 

BNSDOCID; <EP 0466277A1_L> 



EP 0 466 277 A1 



2>t-J <,!(q-l)!(2,+ 2) 

2Jp-a p!(p-i)!(2p-l-2) 

(2p-l)!(2q-l)! 



70 where p and q are integers. Therefore we obtain 



if (p = 0)A(g = 0) 
for (p = 0) A(g > 1) 
for (p> l)A(g = 0) 
for (p > 1)A(? > 1). 



(37) 







1, 




15 










f^L,0,2 — 




A? 
4 ' 



(38) 



20 Similarly, if the camera is set to the second set of camera parameter values Bgz = {>^2,S2,f2,£>2 } with 
the corresponding blur circle radius denoted by A2, then we obtain 



25 



'12.0.0 — 


1, 


'^2.0,1 = 




^^2,0, 2 = 


/t2,2.0 = 



A' 



etc. 



(39) 



30 



35 



40 



Substituting the above values into equations (30) we obtain 



^1,0.0 

'U^l.0,2 = '^1,2,0 = 
2,0.2 = ■^2.2.0 = 



8 

8 ' 



(40) 



45 



50 



Substituting the results of equation (40) into equation (24) and equation (25) and setting A/ = 2 we obtain 



r 0,0 / 0.2 . «2.0\ J 

/n = 9r^l g- [9r.l +^nl ) 

r 0.0 ^? / 0.2 . ^2.0\ 

In = 9r^2 ^ [9ri2 + 5n2 j ' 



(41) 
(42) 



Using equation (32), the focused image can be directly expressed in terms of the set of camera 
parameter values Cai, ea2, and the distance ro of the object as 



55 
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Equating the right hand sides of equations (43) and (44) we obtain 

70 



75 



0.0 Dj / 1 1 1 V ( n"-^ M n^-°\ 



The above equation has the same form as that of equation (8). In the above, equation, the distance To of 
20 the object is the only unknown variable. Therefore equation (45) constitutes one equation in one unknown. 
This equation can be solved to obtain a first set of solutions for the distance To of the object. 
One method of solving equation (45) is described below. From equation (32), we obtain 



25 



1 ^ 1 l^ 



(45) 



Similarly, for the second camera setting specified by Bbz we obtain 

30 



1 /'ZA, . 1 1^ 



(46) 



35 

Equating the right hand side of equation (45) to the right hand side of equation (46) and simplifying, we 
obtain 

A2 = aAi + j8 (47) 

40 

where 

D\ M , ^ Dl (I 11 1\ , . 

The values of a and j8 can be calculated immediately from the set of camera parameters Oai and 632- 
Therefore equation (45) can be written as 

50 

0.0 A? / 2,0 , 0.2\ 0.0 (g Ai / 2.0 , 0.2\ 

?nl - -If V^nl + ^nl j = ffnJ g \9r^2 + ^n2 j • K"^^) 

65 The above equation is a quadratic equation In Ai which can be written in the form 
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10 



15 



20 



25 



30 



35 



50 



55 



a +6 Ai +c = 0 (50) 



where 



a = 



6 = \a0 {gl° + g'J) and (52) 

If the two solutions for Ai are denoted by At and A] , they can be obtained as 



= — and A, = — (54) 

Corresponding to each solution for Ai one solution can be obtained for distance r© from equation (45) 

as 

The first set of solutions is then given by 
Tsi = {rii/12}. (57) 



To find a unique solution for the distance To of the object, a third image g3(x,y) is used to find a second set 
of solutions fsi = {rii,r22} as described earlier. The element that is common to both ^s^ and is the 

40 actual solution for the distance of the object from the camera. 

In the method of autofocusing, first the distance of the object to be focused is determined by using the 
above method of determining the distance fo. Based on distance To and an image focusing formula, a set of 
camera parameter values est = {\f,Sf,ff,Di} is determined which corresponds to focusing the desired object. 
An example of the image focusing formula is the well-known lens formula ^ff = Mu + \lv (see Fig. 2) 

45 which relates the focal length of the camera to the position of the object and its focused image. The relation 
between this image focusing formula and the set of camera parameters Bar can be specified by requiring 
that the radius of the corresponding blur circle be equal to zero. If Af denotes the radius of the blur circle 
corresponding to the camera setting specified by Oaf and To is the distance of the object, then B^f can be 
easily determined by setting Af = 0 in the expression for Af as below: 



ii (1 _ 1 _ iUo. 

2 A, V// ^0 Sf ) 



(58) 



In practice the camera parameters ff and Sf are both free parameters. The value of one of these parameters 
is first fixed and then equation (58) is solved for the other parameter. The camera is then adjusted so that 
the camera parameters are equal to the values determined by ear- This accomplishes autofocusing. 
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In the method of obtaining the focused innage of an object fronn its blurred innage, first the distance fo of 
the object is determined using the method described above. Having determined the distance ro, the focused 
image of the object is computed using either equation (43) or equation (44). This method of computing the 
focused Image is in fact equivalent to first computing the weight parameters using equations (40). then 
5 computing the image derivatives 

9ni (or ) 

10 and substituting the resulting values in the deconvolution formula given by equation (24) (or equation (25)). 
8 Generalization of the Method 

In the method of determining distance described above, note that the distance ro is just one of the 
75 parameters along with the other camera parameters s,f,D,\ on which the point spread function depends. 
From a mathematical point of view, all these parameters including /b play the same role in equation (8). 
There is no speciality associated with any of them. Therefore, a method that is quite similar to the above 
method can be used to find any other parameters s,/,D,X on which the point spread function depends. For 
example, to determine the focal length f of the camera system, the distance /b is taken to be given and 
20 equation (8) is solved for f as before. 

The above method for determining one unknown parameter can be extended to obtain a method to 
determine more than one unknown. In this case we need more than two images. In general, if n unknowns 
are involved then n -h 1 images are required. For example, if we need to determine two parameters, say ro 
and f. then we use three pictures 9^(x,^,g2(x,y) and g3(x,3/) taken with camera settings ea,. Oaa, and ea3 
25 respectively where 

eai = (Si,Di.Xi) (59) 

ea2 = (S2.Cb.X2) (60) 

30 

ea3 = (Sa.Da.Xa). (61) 

The following two equations which are similar to equation (8) can be written for the three images: 

35 

^i.^fc-i C^ieoii/jT-o) ^ni"* = HX! ^2.i.fc-i {^,e^2,/,ro) 9)^2" (62) 

^2,£.Jk-i (^,ea2,/,^o) 9n2~' = ^3,i,k-i ( A, CaS, /, r©) (63) 

k=Oi~Q A:ss:Ois=0 

Equations (62),(63) are solved simultaneously to obtain the two unknowns /,ro. Methods similar to the earlier 
45 case can be used to solve the equations. 

The two equations above imply that the unknowns ro and f should remain the same for all the three 
pictures. However this need not be the case as long as the change in their values are known. For example, 
let the focal lengths be ^ + 61, ^ + 52, and ^ + 63 for the three images where the change in values 5i. 62. 
and 63 are all known. Then, 

50 

^ fc k 

HX] (^tCai,/ 4- Ji,ro) ^Ijf"* = ^2,i.Jt-i (^7ea2,/ + ^2,^0) (64) 

ib=Ot=0 k=Oi=0 

'^7,i,k-i(fi,e^2.f + S2,ro) gl^2'^ = ^3,i,Jfc-i C^, €^3, / -h ^3, Tq) ^^J"' . (65) 

Jfe=Oi=0 fc=Ot=0 
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The two equations above involve only two unknowns (Xand ro) and hence can be solved as before. If more 
than one solution is obtained, then additional images can be used to eliminate spurious solutions and a 
unique solution can be obtained. 

After determining the unknown parameters, the deconvolution formula (24) can be used to obtain the 
5 focused image. 

A camera is a particular example of a larger class of instruments which can be modeled as linear shift- 
invariant systems. The method of determining the unknown camera parameters using a set of images taken 
with different camera parameter values can be generalized to all linear shift-iri variant systems. The input 
signal and the output signals of the linear shift-invariant system can be multi-dinnensional signals. A general 

10 linear shift-invariant system is characterized by a set of system parameters just as a camera is 
characterized by a set of camera parameters. A set of output signals of the linear shift-invariant system are 
obtained for different values of the system parameters. A pair of output signals can be used to formulate 
one equation similar to equation (8). At least as many such equations as the number of unknown system 
parameters are formulated for different pairs of output signals. The resulting equations can be solved to 

15 determine all the unknown system parameters. Then a deconvolution formula similar to equation (24) can be 
used to obtain the input signal to the linear shift- in variant system. The relevant deconvolution formula is 
disclosed by this same inventor in another patent application filed recently: "A Method and Apparatus for 
Restoring Convolution Degraded Signals", U.S. patent application filed on July 3, 1990. 

The general method outlined above could be useful in a wide variety of instruments. A camera system 

20 is just one example of an instrument where the application of this method has been presented in complete 
detail. 

9 Conclusion, Ramifications, and Scope of Invention 

25 The present invention was mainly conceived in the context of machine vision applications such as robot 

vision, autonomous vehicle navigation, finding approximate distance of objects as part of a stereo vision 
system, etc. There are also other applications where this invention can be used after minor modifications. 
For example it is useful in television microscopy in obtaining improved focus images of three-dimensional 
specimens. Another application is a completely electronic camera where taking a picture involves storing 

30 the image signal directly on a magnetic disk for two or more camera settings, and then processing this 
image signal on a computer in conjunction with camera characteristics to obtain pictures with desired depth- 
of-field. 

While the description in this report of the method, apparatus, and applications contains many specific- 
ities, these should not be construed as limitations on the scope of the invention, but rather as exemplifica- 
35 tions of preferred embodiments thereof. Further modifications and extensions of the present invention herein 
disclosed will occur to persons skilled in the art to which the present invention pertains, and all such 
modifications are deemed to be within the scope and spirit of the present invention as defined by the 
appended claims and their legal equivalents thereof. 
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45 Claims 

1. A method of determining the distance of an object from a camera, said camera characterized by a set 
of camera parameters, said method comprising: 

(a) setting said camera parameters to a first set of camera parameter values, forming an image of 
50 said object inside said camera, and recording a first image; 

(b) setting said camera parameters to a second set of camera parameter values, forming an image 
of said object inside said camera, and recording a second image; 

(c) preprocessing said first image to obtain a first normalized image such that, said first normalized 
image is equal to the convolution of a focused image of said object with a first point spread function 

55 of said camera characterized by said first set of camera parameter values; 

(d) preprocessing said second image to obtain a second normalized image such that said second 
normalized image is equal to the convolution of a second point spread function of said camera 
characterized by said second set of camera parameter values; 
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(e) computing a set of derivatives of said first normalized innage to obtain a first set of image 
derivatives; 

(f) computing a set of derivatives of said second normalized image to obtain a second set of Image 
derivatives; 

5 (g) substituting the values of said first set of camera parameter values, said second set of camera 

parameter values, said first set of image derivatives and said second set of image derivatives, into 
an equation obtained by equating a first expression for said focused image in terms of said first set 
of camera parameter values and said first set of image derivatives to a second expression for said 
focused image expressed in terms of said second set of camera parameter values and said second 

70 set of image derivatives thereby obtaining an equation in which the distance of said object is the 

only unknown variable; 

(h) solving the equation obtained in step (g) to obtain a first set of solutions for the distance of said 
object, whereby no Fourier transforms are computed in solving for distance; 

(i) setting said camera parameters to a third set of camera parameter values, forming an image of 
J 5 said object inside said camera, and recording a third image; and 

(j) determining the member of said first set of solutions for distance obtained in step (h) which is 
consistent with said third set of camera parameter values and said third image to obtain the distance 
of said object from said camera system. 

20 2. The method of claim (1), wherein said camera includes an Image detector for recording images, a 
camera aperture through which light enters, an optical system characterizable by a first principal plane 
and a second principal plane, said second principal plane being closer to said image detector than said 
first principal plane, and a light filter. 

25 3. The. method of claim (2), wherein said set of camera parameters includes at least one of 

(a) the distance between, said second principal plane and said image detector, 

(b) the diameter of said camera aperture, 

(c) the focal length of said optical system, and 

(d) the spectral light transmission characteristics of said light filter. 

30 

4. The method of claim (3). wherein the preprocessing operations in steps (c) and (d) include at least one 
of the operations of 

(a) correcting for the sensor response characteristic of said image' detector, 

(b) correcting for the non-uniform light transmission of said optical system. 
35 (c) magnification normalization of said first image and said second image, 

(d) brightness normalization of said first image and said second image, and 

(e) noise filtering to reduce the effects of noise. 

5. The method of claim (2), wherein said first expression in step (g) is the result of multiplying each 
40 member of said first set of image derivatives by a weight parameter to obtain a first set of weighted 

image derivatives and summing all members of said first set of weighted image derivatives. 

6. The method of claim (2), wherein said second expression in step (g) is the result of multiplying each 
member of said second set of image derivatives by a weight parameter to obtain a second set of 

45 weighted image derivatives and summing all members of said second set of weighted image 

derivatives. 

7. The method of claim (5), wherein each said weight parameter depends on a first set of moment 
parameters which together characterize said first point spread function (in step (c)). 

50 

8. The method of claim (6), wherein each said weight parameter depends on a second set of moment 
parameters which together characterize said second point spread function (In step (d)). 

9. The method of claim (7), wherein said first point spread function is symmetric and therefore all odd 
55 moment parameters in said first set of moment parameters are all zero. 

10. The method of claim (10), wherein said first point spread function has a non-zero constant value inside 
a region having the same shape as that of said camera aperture except for a magnification factor and 
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has a zero value outside of said region. 

11. The method of claim (10), wherein said camera aperture has a circular shape and therefore said first 
point spread function has a non-zero constant value inside a circular region and has a zero value 

5 outside of said circular region. 

12. The method of claim (1), wherein step (j) includes the following steps 

(I) preprocessing said third image to obtain a third normalized image which is equal to the 
convolution of a third point spread function of said camera characterized by said third set of camera 
10 parameter values; 

(m) computing a set of derivatives of said third normalized image to obtain a third set of image 
derivatives; 

(n) substituting the values of said second set of camera parameter values, said third set of camera 
parameter values, said second set of image derivatives, and said third set of image derivatives, into 
15 an equation obtained by equating said second expression to a third expression for said focused 

image expressed in terms of said third set of camera parameter values and said third set of image 
derivatives thereby obtaining an equation in which the distance of said object is the only unknown 
variable; 

(0) solving the equation obtained in step (n) to obtain a second set of solutions for the distance of 
20 said object; and 

(p) intersecting said first set of solutions and said second set of solutions to obtain a common 
solution which gives the distance of said object from said camera. 

13, A method of autofocusing a camera to an object, said camera characterized by a set of camera 
25 parameters, said method comprising: 

(a) setting said camera parameters to a first set of camera parameter values, forming an image of 
said object inside said camera, and recording a first image; 

(b) setting said camera parameters to a second set of camera parameter values, forming an image 
of said object inside said camera, and recording a second image; 

30 (c) preprocessing said first image to obtain a first normalized image such that, said first normalized 

image is equal to the convolution of a focused image of said object with a first point spread function 
of said camera characterized by said first set of camera parameter values; 

(d) preprocessing said second image to obtain a second normalized Image such that said second 
normalized image is equal to the convolution of a second point spread function of said camera 

35 characterized by said second set of camera parameter values; 

(e) computing a set of derivatives of said first normalized image to obtain a first set of image 
derivatives; 

(f) computing a set of derivatives of said second normalized image to obtain a second set of image 
derivatives; 

40 (g) substituting the values of said first set of camera parameter values, said second set of camera 

parameter values, said first set of image derivatives and said second set of image derivatives, into 
an equation obtained by equating a first expression for said focused image in terms of said first set 
of camera parameter values and said first set of image derivatives to a second expression for said 
focused image expressed in terms of said second set of camera parameter values and said second 

45 set of image derivatives thereby obtaining an equation in which the distance of said object is the 

only unknown variable; 

(h) solving the equation obtained in step (g) to obtain a first set of solutions for the distance of said 
object, whereby no Fourier transforms are computed in solving for distance; 

(1) setting said camera parameters to a third set of camera parameter values, forming an image of 
50 said object inside said camera, and recording a third image; 

(j) determining the member of said first set of solutions for distance obtained in step (h) which is 
consistent with said . third set of camera parameter values and said third image to obtain the distance 
of said object from said camera system; and 

(k) computing a new set of camera parameter values for focusing said object based on the distance 
55 of said object determined in step (j) and setting said set of camera parameters to said new set of 

camera parameter values thereby autofocusing said camera to said object. 

14, The method of claim (13), wherein said camera includes an image detector for recording images, a 



21 



BNSDOCID: <EP 0466277A1 J_> 



EP 0 466 277 A1 



camera aperture through which light enters, an optical system characterizable by a first principal plane 
and a second principal plane, said second principal plane being closer to said image detector than said 
first principal plane, and a light filter. 

5 15. The method of claim (14). wherein said set of camera parameters includes at least one of 

(a) the distance between said second principal plane and said image detector, 

(b) the diameter of said camera aperture, 

(c) the focal length of said optical system, and 

(d) the spectral light transmission characteristics of said light filter. 

70 

16. The method of claim (15), wherein the preprocessing operations in steps (c) and (d) include at least 
one of the operations of 

(a) correcting for the sensor response characteristic of said image detector, 

(b) correcting for the non-uniform light transmission of said optical system, 
76 (c) magnification normalization of said first image and said second image, 

(d) brightness normalization of said first image and said second image, and 

(e) noise filtering to reduce the effects of noise. 

17. The method of claim (14), wherein said first expression in step (g) is the result of multiplying each 
20 member of said first set of image derivatives by a weight parameter to obtain a first set of weighted 

image derivatives and summing all members of said first set of weighted image derivatives. 

18. The method of claim (14). wherein said second expression in step (g) is the result of multiplying each 
member of said second set of image derivatives by a weight parameter to obtain a second set of 

25 weighted image derivatives and summing all members of said second set of weighted image 
derivatives. 

19. The method of claim (17), wherein each said weight parameter depends on a first set of moment 
parameters which together characterize said first point spread function (in step (c)). 

30 

20. The method of claim (18), wherein each said weight parameter depends on a second set of moment 
parameters which together characterize said second point spread function (in step (d)). 

21. The method of claim (19), wherein said first point spread function is symmetric and therefore all odd 
35 moment parameters in said first set of moment parameters are all zero. 

The method of claim (22), wherein said first point spread function has a non-zero constant value inside 
a region having the same shape as that of said camera aperture except for a magnification factor and 
has a zero value outside of said region. 

The method of claim (22), wherein said camera aperture has a circular shape and therefore said first 
point spread function has a non-zero constant value inside a circular region and has a zero value 
outside of said circular region. 

The method of claim (13), wherein step Q) includes the following steps 

(I) preprocessing said third image to obtain a third normalized image which is equal to the 
convolution of a third point spread function of said camera characterized by said third set of camera 
parameter values; 

(m) computing a set of derivatives of said third normalized image to obtain a third set of image 
derivatives; 

(n) substituting the values of said second set of camera parameter values, said third set of camera 
parameter values, said second set of image derivatives, and said third set of image derivatives, into 
an equation obtained by equating said second expression to a third expression for said focused 
image expressed in terms of said third set of camera parameter values and said third set of image 
derivatives thereby obtaining an equation in which the distance of said object is the only unknown 
variable; 

(o) solving the equation obtained in step (n) to obtain a second set of solutions for the distance of 
said object; and 
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(p) intersecting said first set of solutions and said second set of solutions to obtain a common 
solution which gives the distance of said object from said camera. 

25. A method of obtaining the focused image of an object from a blurred image of said object, said blurred 
5 image formed in a camera, said camera characterized by a set of camera parameters, said method 

comprising; 

(a) setting said camera parameters to a first set of camera parameter values, forming an image of 
said object inside said camera, and recording a first image; 

(b) setting said camera parameters to a second set of camera parameter values, forming an image 
10 of said object inside said camera, and recording a second image; 

(c) preprocessing said first image to obtain a first normalized image such that, said first normalized 
image is equal to the convolution of a focused image of said object with a first point spread function 
of said camera characterized by said first set of camera parameter values; 

(d) preprocessing said^second image to obtain a second normalized image such that said second 
75 normalized image is equal to the convolution of a second point spread function of said camera 

characterized by said second set of camera parameter values; 

(e) computing a set of derivatives of said first normalized image to obtain a first set of image 
derivatives; 

(f) computing a set of derivatives of said second normalized image to obtain a second set of image 
20 derivatives; 

(g) substituting tile values of said first set of camera parameter values, said second set of camera 
parameter values, said first set of image derivatives and said second set of image derivatives, into 
an equation obtained by equating a first expression for said focused image in terms of said first set 
of camera parameter values and said first set of image derivatives to a second expression for said 

25 Jocused image expressed in terms of said second set of camera parameter values and said second 

set of image derivatives thereby obtaining an equation in which the distance of said object is the 
only unknown varialDle; 

(h) solving tile equation obtained in step (g) to obtain a first set of solutions for the distance of said 
object, whereby no Fourier, transforms are computed in solving for distance; 

30 (i) setting said camera parameters to a third set of camera parameter values, forming an image of 

said object inside said camera, and recording a third image; 

G) determining the member of said first set of solutions for distance obtained in step (h) which is 
consistent with said third set of camera parameter values and said third image to obtain the distance 
of said object from said camera system; and 
35 (k) substituting the values of said first set of camera parameter values, said first set of image 

derivatives, and the distance of said object from said camera determined in step (j), into said first 
expression to obtain the focused image of said object. 

26. Apparatus for determining the distance of an object from a camera, said apparatus comprising a 
40 camera characterized by a set of camera parameters, a camera controller for setting said set of camera 

parameters to desired values, an image normalization means for normalizing different images recorded 
with different values for said set of camera parameters, a derivative computation means for computing 
derivatives of normalized images, an equation solving means which can substitute into an equation the 
values of different derivatives of normalized images and camera parameter values and solve for the 
45 distance, of said object from said camera system, and a set intersection means for intersecting two sets 
of solutions for distance and determining the common solution. 

27. The apparatus of claim (26), which further includes a focus parameter determination means for 
determining a value for said set of camera parameters based on the distance of said object from said 

50 camera so that said object is focused by setting said set of camera parameters to the value determined 

by said focus parameter determination means. 

28. The apparatus of claim (26), which further includes a focused image determination means which takes 
as input a set of derivatives of a blurred image of said object, values of said set of camera parameters, 

55 and distance of said object from said camera, and uses a deconvolution formula to give as output the 

focused image of said object without computing any Fourier transforms. 
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